Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System tests should cleanup after themselves #2216

Merged
merged 77 commits into from
Oct 21, 2024
Merged

System tests should cleanup after themselves #2216

merged 77 commits into from
Oct 21, 2024

Conversation

ATorrise
Copy link
Contributor

@ATorrise ATorrise commented Aug 6, 2024 •
edited
Loading

What It Does
This PR enhances the test framework to simplify and automate the cleanup of test resources, such as datasets, USS files, jobs, and local files. The TestEnvironment, ITestEnvironment, and TestUtils classes have been updated to track and delete resources after system tests, preventing any leftover artifacts on the mainframe or local filesystem.

How to Test : Run System Tests

  • Execute the system tests as usual. Verify that all created resources (datasets, USS files, jobs, local files) are automatically cleaned up after the tests complete.
  • Ensure that no traces of jobs, USS files, or datasets are left behind on the mainframe.

Review Checklist
I certify that I have:

Additional Comments

  • Test Stability
    • Occasionally, environment-specific issues may arise, especially with test datasets like IEFBR14 files. I'm not sure how to rectify this and definitely need areas using these files double-checked.
  • Unchanged Files:
    • zowe-cli\packages\zosjobs\__tests__\__system__\MonitorJobs.system.test.ts only had minor indentation changes and was left intact as it leaves no jobs behind on the mainframe.
    • If a test file cleans up all resources, it was left without changes to avoid unnecessary modifications since most of these test files are large and would require many lines of renovations.

@ATorrise ATorrise changed the base branch from master to next August 6, 2024 13:33
@ATorrise ATorrise linked an issue Aug 6, 2024 that may be closed by this pull request
TD(tests): Make sure that system tests cleanup after themselves #2178
Closed
@codecov Codecov
Copy link

codecov bot commented Aug 6, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.08%. Comparing base (a1461ca) to head (cacdd3a).
Report is 78 commits behind head on master.

Additional details and impacted files 
@@           Coverage Diff            @@
##           master    #2216    +/-   ##
========================================
  Coverage   91.08%   91.08%            
========================================
  Files         628      628            
  Lines       17896    17896            
  Branches     3847     3741   -106     
========================================
  Hits        16300    16300            
  Misses       1595     1595            
  Partials        1        1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ATorrise ATorrise force-pushed the system-test-cleanup branch from 0f61785 to 4f059e6 Compare August 16, 2024 15:29
@ATorrise
modifying upload.system.test took the longest... removing some test f…
81048a7 
…iles from permanent residency, ,instead creating them from scratch everytime

Signed-off-by: ATorrise <[email protected]>
@ATorrise
took awhile to see the difference in regex between v1 and v2 that i w…
a505c0d 
…as matching for this test. also took a while to identify which shell script was populating 2 jobs

Signed-off-by: ATorrise <[email protected]>
@ATorrise
providing more clarity in a comment about calculating previous jobid
56f0e9a 
Signed-off-by: ATorrise <[email protected]>
@ATorrise
fixing cli.zos-jobs.download and its accompanying scripts
c458433 
Signed-off-by: ATorrise <[email protected]>
@ATorrise ATorrise marked this pull request as ready for review August 26, 2024 14:58
github-advanced-security[bot]
github-advanced-security bot found potential problems Oct 11, 2024
View reviewed changes
packages/cli/__tests__/zosjobs/__system__/cancel/cli.zos-jobs.cancel.job.system.test.ts Fixed Show fixed Hide fixed
packages/cli/__tests__/zosjobs/__system__/cancel/cli.zos-jobs.cancel.job.system.test.ts Fixed Show fixed Hide fixed
packages/cli/__tests__/zosjobs/__system__/cancel/cli.zos-jobs.cancel.job.system.test.ts Fixed Show fixed Hide fixed
packages/cli/__tests__/zosjobs/__system__/cancel/cli.zos-jobs.cancel.job.system.test.ts Fixed Show fixed Hide fixed
packages/cli/__tests__/zosjobs/__system__/cancel/cli.zos-jobs.cancel.job.system.test.ts Fixed Show fixed Hide fixed
packages/cli/__tests__/zosjobs/__system__/cancel/cli.zos-jobs.cancel.job.system.test.ts Fixed Show fixed Hide fixed
packages/cli/__tests__/zosjobs/__system__/cancel/cli.zos-jobs.cancel.job.system.test.ts Fixed Show fixed Hide fixed
github-advanced-security[bot]
github-advanced-security bot found potential problems Oct 14, 2024
View reviewed changes
packages/cli/__tests__/workflows/__system__/create/cli.workflows.create.common.system.test.ts Fixed Show fixed Hide fixed
packages/cli/__tests__/workflows/__system__/create/cli.workflows.create.common.system.test.ts Fixed Show fixed Hide fixed
packages/cli/__tests__/workflows/__system__/create/cli.workflows.create.common.system.test.ts Fixed Show fixed Hide fixed
packages/cli/__tests__/workflows/__system__/create/cli.workflows.create.common.system.test.ts Fixed Show fixed Hide fixed
packages/cli/__tests__/workflows/__system__/create/cli.workflows.create.common.system.test.ts Fixed Show fixed Hide fixed
packages/zosfiles/__tests__/__system__/methods/delete/DeleteUSSFile.system.test.ts Fixed Show fixed Hide fixed
packages/zosfiles/__tests__/__system__/methods/delete/DeleteUSSFile.system.test.ts Fixed Show fixed Hide fixed
packages/zosfiles/__tests__/__system__/methods/delete/DeleteUSSFile.system.test.ts Fixed Show fixed Hide fixed
packages/zosfiles/__tests__/__system__/methods/delete/DeleteUSSFile.system.test.ts Fixed Show fixed Hide fixed
packages/zosfiles/__tests__/__system__/methods/delete/DeleteUSSFile.system.test.ts Fixed Show fixed Hide fixed
@t1m0thyj t1m0thyj force-pushed the system-test-cleanup branch from 0faa40b to 336d3e4 Compare October 14, 2024 12:49
@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

@t1m0thyj t1m0thyj requested a review from zFernand0 October 14, 2024 18:59
@t1m0thyj t1m0thyj linked an issue Oct 14, 2024 that may be closed by this pull request
Generalize CLI system tests for jobs to support more environments #2263
Closed
@t1m0thyj t1m0thyj self-assigned this Oct 14, 2024
@t1m0thyj t1m0thyj requested review from t1m0thyj and gejohnston and removed request for t1m0thyj October 14, 2024 19:40
@t1m0thyj t1m0thyj dismissed their stale review October 14, 2024 19:41

Requested changes addressed

@t1m0thyj t1m0thyj removed their request for review October 14, 2024 19:41
awharn
awharn approved these changes Oct 18, 2024
View reviewed changes
Copy link
Member

@awharn awharn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, and the tests seem to pass. Some jobs are removed by job name and others are removed by job id. It would be nice if it was consistent, but since removing them by name can remove large quantities at once, and the PR seems to use that behavior, I think it should be fine to leave this way. :)

zFernand0
zFernand0 approved these changes Oct 21, 2024
View reviewed changes
Copy link
Member

@zFernand0 zFernand0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all the great work on this! 🙏

LGTM! 😋

@zFernand0 zFernand0 merged commit 027d9cc into master Oct 21, 2024
18 checks passed
@zFernand0 zFernand0 deleted the system-test-cleanup branch October 21, 2024 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@t1m0thyj t1m0thyj t1m0thyj left review comments

@awharn awharn awharn approved these changes

@zFernand0 zFernand0 zFernand0 approved these changes

@traeok traeok Awaiting requested review from traeok

@gejohnston gejohnston Awaiting requested review from gejohnston

Assignees

@t1m0thyj t1m0thyj

@ATorrise ATorrise

Labels
None yet
Projects
Zowe CLI Squad
Status: Closed
Milestone
No milestone
5 participants
@ATorrise @awharn @t1m0thyj @gejohnston @zFernand0